VLDeformer: Vision–Language Decomposed Transformer for fast cross-modal retrieval

Cross-model retrieval has emerged as one of the most important upgrades for text-only search engines (SE). Recently, with powerful representation pairwise text–image inputs via early interaction, accuracy vision–language (VL) transformers outperformed existing methods retrieval. However, when same paradigm is used inference, efficiency VL still too low to be applied in a real cross-modal SE. Inspired by mechanism human learning and using knowledge, this paper presents novel Vision–Language Decomposed Transformer (VLDeformer), which greatly increases while maintaining their outstanding accuracy. By proposed method, cross-model separated into two stages: transformer stage, decomposition stage. The latter stage plays role single modal indexing, some extent like term indexing text model learns knowledge from early-interaction pre-training then decomposed an individual encoder. VLDeformer requires only small target datasets supervision achieves both 1000+ times acceleration less than 0.6% average recall drop. also outperforms state-of-the-art visual-semantic embedding on COCO Flickr30k.1